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(54) Method and system for forming skeletons for generating verification systems 



(57) A verification system for a procedure interface 
is generated by using formal specifications of the pro- 
cedure interface and generating test suites. The test 
suites are generated from the formal specifications and 
templates or skeletons which are used to generate an 
element of a verification system. The skeletons are gen- 
erated based on decomposition of test suits. 
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Description 

[0001] This invention relates to a system and method 
for forming skeletons for generating a verification sys- 
tem for verifying procedure interfaces. 5 
[0002] A software system contains a functionally 
closed set of procedures. In order to ensure correct im- 
plementation of the software system, it is desirable to 
determine a software contract, i.e., elements and func- 
tional specifications of external interfaces of the soft- io 
ware system, and carry out conformance testing of the 
software contract implementation. Since the elements 
of the software contract are procedures, it is, in fact, Ap- 
plication Programming Interface (API) testing. 
[0003] A kernel of an Operating System (OS) com- is 
prises API. For example, a Support Operating System 
(SOS) is a real-time OS for a Digital Multiplexing Switch 
(DMS) for a communication system. SOS comprises a 
plurality of processes for supporting the operation of 
DMS. The lowest layer of SOS is SOS kernel. The SOS 20 
kernel allocates resources to the processes running un- 
der SOS. The SOS kernel also provides communication 
among these processes. The SOS kernel also creates, 
controls and removes these processes. 
[0004] SOS supports more than 25 million lines of 25 
code for applications and utilities. Thus, it is critical that 
user procedure interfaces of the SOS kernel be stable 
and reliable for correct performances of the DMS switch. 
The SOS kernel consists of over 1,700 procedures or 
over 230,000 lines of source code. Thus, it is very com- 30 
plicated and time consuming processes to generate a 
system for verifying such complex procedure interfaces. 
There existed no automatic or semi-automatic mecha- 
nisms to aid such generation of a verification system. 
[0005] At the same time, SOS continuously evolves. 35 
Also, SOS is often ported to new hardware and software 
platforms. While more than 75% of the kernel proce- 
dures are machine-independent, the remainder of the 
kernel procedures are very machine dependent. The re- 
mainder describes particularity of memory, interproces- 40 
sor communication and communication with peripheral 
devices. Accordingly, when SOS evolves or SOS is 
ported to a new platform, the SOS kernel and its proce- 
dure interfaces are also modified. Thus, the verification 
system for the procedure interfaces of the SOS kernel 45 
also needs to be modified. However, there existed no 
automatic or semi-automatic modifying mechanisms to 
aid such modifications. 

[0006] There are some systems proposed for building 
a verification process. One of such systems is Interac- so 
five Tree and Tabular Combined Notation (TTCN) Editor 
and executor (ITEX). ITEX is a test environment for 
communicating systems. It includes a TTCN and Ab- 
stract Syntax Notation. 1 (ASN.l) analysis and design 
tool, a test simulator and support for generation of com- 55 
plete Executable Test Suites (ETS). In accordance with 
ITEX, a Test Suite is made up of Test Cases in form of 
tables. ITEX provides a set of highly integrated tools for 



development and maintenance of Abstract Test Suites 
(ATS) written in TTCN. ITEX supports phases of the test 
suite development including Test Case Generation, Ed- 
iting, Verification Validation and Execution. This toolset 
is integrated with the Specification Description Lan- 
guage (SDL) Design Tool (SDT), which is an environ- 
ment for design of SDL specifications. Test suites de- 
scribed with TTCN can be transformed to the form that 
allows testing both implementation in some program- 
ming language and specification in SDL. However, this 
approach is unsuitable for API testing. TTCN does not 
permit declaration of pointers and other software entities 
that do not have textual (literal) representation. A major 
limitation of SDL-like specifications is their explicit form. 
This means that it is easy to build models and prototypes 
based on them but it is very difficult to develop a system 
of constraints that define the union of all possible imple- 
mentations. 

[0007] Another example is the Algebraic Design Lan- 
guage (ADL)/ADL2. From formal specifications, ADL 
generates test oracles and skeletons for building test 
drivers and documentation. ADL uses not one of the 
popular specification languages but extensions of C and 
C++ languages. There are ideas on extensions of Java 
and other object-oriented languages aimed at develop- 
ing software in "Design-by-Contract" fashion. However, 
despite the obvious advantages of better acceptance of 
such languages in the software engineering community, 
the concept, not to mention the common notation, is still 
far in the future. ADL has a limited range of API classes 
for which it can provides means for specifications and 
automatic test generation. ADL provides adequate tools 
for test generation automation only for procedures 
whose parameters allow independent enumeration and 
allows testing procedures one by one. This means that 
ADL omits procedures with dependent parameters, pro- 
cedures that require testing in a group, e.g., "open- 
close", or those that require testing in parallel mode, e. 
g., "lock-unlock", or "send-receive". 
[0008] Another example is formal derivation of Finite 
State Machines (FSM) for class testing proposed by L. 
Murray, D. Carrington, I. MacCol), J. McDonald and P. 
Strooper in "Formal Derivation of Finite State Machines 
for Class Testing", in Jonatthan P. Bowen, Andreas Fett, 
Michael G. Hinchey (eds.) ZUM'98: The Z Formal Spec- 
ification Notation. 11-th International Conference of Z 
Users, Berlin, Germany, Sept. 1998, Proceeding, Lec- 
ture Notes in Computer Science, v 1493, pp. 42-59. 
This work is at the research stage. The authors propose 
a scheme for organization of procedure group testing 
using Object-Z as specification language and C++ as 
programming language. The task of this work is stated 
to build test suites to verify conformance of the imple- 
mentation to the specification using formal specifica- 
tions of the methods for a class. As a test coverage cri- 
terion, the union of twocriteria is used: tocoverall equiv- 
alency classes that represent the areas obtained as a 
result of partition analysis, and then, to check results on 
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or near the boundaries. However, the authors of this 
work do not try to solve the problem of complete auto- 
mation of test generation. Nor do they attempt to support 
any elements of the preparation phase with tools. Par- 
tition and boundary analysis is done manually according 
to the methodology proposed by the authors. In a similar 
way, they build the specification of oracles. Oracles, 
once compiled into C++, call target procedures and ver- 
ify the conformance of the results to the specifications. 
This testing scheme is a framework that dynamically 
generates test sequences of procedure calls. The 
framework is controlled by the FSM description which 
represents an abstraction of a state transition graph of 
the test class. The authors describe the methodology of 
building specifications for the classes of states and tran- 
sitions between them while considering the problem of 
exclusion of inaccessible states. 

[0009] This approach needs the full description of the 
FSM that models the states of the system under test. 
The theoretical weakness of this approach is that it does 
not try to come up with a formal methodology to build 
transformation specifications. It is obvious that serious 
problems will be encountered when attempting to apply 
this approach to specifications of real-life complexity. In 
practical sense, it is clear that the process of test deri- 
vation from the specifications is mostly manual activity 
which limits its applicability to industrial software. 
[0010] The development of verification systems is a 
labour-intensive process. The total size of a verification 
system is often similar to the size of the software under 
test. 

[0011] Currently, the most commonly used approach 
to the verification system development is the manual de- 
velopment. There are some tools that can generate 
some elements of a verification system for the most sim- 
ple cases of the test parameter set generation, namely, 
for the case of the manually written list of the test pa- 
rameter sets in literal form. However, this approach can 
not be used if some parameter could not be written in 
literal form. For example, a UNIX file descriptor is re- 
turned by operating system as a result of the system call 
"open file", and can not be written in literal form. This 
approach also can not be used for testing of the proce- 
dure sequence when the out-parameter of some proce- 
dure is used as the in-parameter of the another proce- 
dure. 

[001 2] It is therefore desirable to provide means to de- 
crease the efforts of the verification system develop- 
ment and to increase its reliability. 
[0013] The present invention provides a method and 
system for forming templates or skeletons which are 
used to generate an element of a verification system. 
[001 4] In accordance with an aspect of the present in- 
vention, there is provided a method for forming a skel- 
eton tool usable for generating a test suite for a verifi- 
cation system for verifying a Procedure Interface Under 
Test (PIUT). The method comprises decomposing ex- 
isting test suites, the test suites having automatically 



generated components and manually developed com- 
ponents and being written in its test suite implementa- 
tion language; defining one or more standard schemes 
of procedure testing based on the decomposition of the 

5 test suites; providing skeleton description for each 
scheme in skeleton definition language; and transform- 
ing the skeleton description for each scheme into a skel- 
eton tool for generating the test suite of the scheme. 
[0015] In accordance with another aspect of the 

io present invention, there is provided a system for forming 
a skeleton tool usable for generating a test suite for a 
verification system for verifying a Procedure Interface 
under Test (PIUT). The system comprises a decompos- 
er, a skeleton describer and a skeleton transformer. The 

15 decomposer is provided for decomposing test suites. 
The test suites have automatically generated compo- 
nents and manually developed components. They are 
written in its test suite implementation language. The de- 
composed test suites are used to define one or more 

20 standard schemes of procedure testing. The skeleton 
describer is used for providing skeleton description for 
each scheme in skeleton definition language. The skel- 
eton transformer is provided for transforming the skele- 
ton description for each scheme into a skeleton tool for 

25 generating the test suite of the scheme. 

[0016] Other aspects and features of the present in- 
vention will be readily apparent to those skilled in the art 
from a review of the following detailed description of pre- 
ferred embodiments in conjunction with the accompa- 

30 nying drawings. 

[0017] Exemplary embodiment of the present inven- 
tion will be further understood from the following de- 
scription with reference to the drawings in which: 

35 Figure 1 is a diagram showing an example of a ver- 
ification system generator to which the preferred 
embodiment of the present invention can be ap- 
plied; 

Figure 2 is a flowchart showing an embodiment of 
40 a method for generating a verification system; 

Figure 3 is a flowchart showing steps of formal 
specification generation shown in Figure 2; 
Figure 4 is a diagram showing a structure of a Sup- 
port Operation System (SOS); 
45 Figures is a diagram showing an example of formal 
specification generation; 

Figure 6 is a diagram showing an example of test 
suite generation; 

Figure 7 is a flowchart showing a method of forming 
50 a skeleton according to an embodiment of the 
present invention; 

Figure 8 is a diagram showing an example of the 
skeleton description development; 
Figure 9 is a diagram showing a system for forming 
55 skeletons in accordance with an embodiment of the 
present invention; and 

Figure 10 is a block diagram showing an example 
of a system for completing test suite sources. 
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[0018] There are different kinds of API entities, such 
as procedures, operations, functions, methods in C++ 
and subroutines in Fortran. In this specification, these 
terms are considered synonyms and all are called "pro- 
cedure". 5 
[0019] Referring to Figures 1 and 2, an example of a 
verification system generator 1 and a method for gen- 
erating a verification system 2 to which the present in- 
vention may be suitably applied are described. The ver- 
ification system 2 is generated for verifying a procedure 10 
interface 4 of a System Under Test (SUT) 3. 
[0020] The verification system generator 1 comprises 
means 12 for generating formal specifications, a test 
source generator 14 and a repository 16. As shown in 
Figure 2, the means 1 2 for generating formal specifica- 15 
tions generates formal specifications of the procedure 
interface 4 (S10). Based on the formal specifications, 
the test source generator 14 generates test sources 
(S20). The generated formal specifications and the test 
sources are stored in the repository 16(S30). 20 
[0021] The test sources are used to generate a test 
suite 22. The test suite 22 is a set of programs and test 
data intended for the use in verifying the target proce- 
dure interface 4. 

[0022] The formal specifications are generated in a 25 
form independent from implementation of the SUT 3. 
That is, the formal specifications do not depend on the 
implementation language, software or hardware of SUT 
3, as further described later. The test sources that are 
generated based on the implementation independent 30 
formal specifications are also implementation independ- 
ent. Accordingly, the test sources may be used on any 
implementation of the SUT or modified versions of the 
SUT 

[0023] The SUT 3 uses specific implementation Ian- 35 
guage. The test sources are written in specification lan- 
guage that is independent from the implementation lan- 
guage. Accordingly, in order to execute the test sources 
on the SUT 3 to verify the procedure interface 4 of the 
SUT 3, the test sources are translated in language ex- 40 
ecutable on the SUT 3 (S40). The translation is carried 
out by an implementation language compiler 18. The 
compiler 18 compiles some executable subsets of test 
sources in the specification language into programs in 
the implementation language of the SUT 3. The com- 45 
plied programs are specifications in implementation lan- 
guage that can be interpreted as description of some 
algorithms. 

[0024] Thus, the generation of the verification system 
2 is carried out in two stages. First, generation of imple- 50 
mentation independent programs is performed. Then, 
implementation independent programs are compiled in- 
to those in implementation language of the SUT 3. Such 
a two step generation method allows the means 12 for 
generating specifications and the test source generator 55 
14 of the verification system generator 1 to be imple- 
mentation-independent tools and, in particular, imple- 
mentation-language-independent tools. 
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[0025] The complier 18 may be a part of the verifica- 
tion system generator 1 or may be provided separately 
from the verification system generator 1 . 
[0026] The compiled test sources form the test suite 
22. As the test sources are implementation independ- 
ent, the test suite 22 is also independent from the im- 
plementation of the target SUT 3, other than the lan- 
guage used. That is, the test suite 22 does not depend 
on the implementation software or hardware of SUT 3. 
By using the test suite 22, a test harness 20 including 
the test suite 22 and a test bed 24 is formed for verifying 
the procedure interface 4 of the SUT 3, as further de- 
scribed below. 

[0027] The test suite 22 executes tests on the SUT 3 
(S50) and analyses results of the tests to verify the pro- 
cedure interface 4 (S60). 

[0028] The verification system generator 1 "auto- 
mates" test generation of real software for verifying a 
procedure interface 4 of an SUT 3. The expression "au- 
tomation" used herein does not necessarily mean fully 
automated manipulation that creates ready for use test 
data, test sequences and other infrastructure for test ex- 
ecution and test result analysis. An "automated" proc- 
ess may include steps of manually writing some com- 
ponents in implementation language. When the total 
size of such manually developed components is small 
as a whole, the process may be considered "automat- 
ed". 

[0029] It is preferable that the test source generator 
1 4 comprises a test driver generator 30 and a test case 
parameter generator 32. 

[0030] The test case parameter generator 32 gener- 
ates test case parameter sources for generating test 
case parameters. That is, the test case parameter gen- 
erator 32 generates constant arrays and programs that 
generate and select needed test case parameters. The 
test case parameters are represented by these constant 
arrays and programs. 

[0031] Based on the formal specifications, the test 
driver generator 30 generates test driver sources for 
generating test drivers. The test drivers execute tests 
on the SUT 3 using the test case parameters in imple- 
mentation environments and analysing results of tests. 
[0032] The test drivers comprise programs to execute 
and control testing of the procedure interface 4. The test 
case parameters are parameters of a test case. A test 
case is an instance of a tested procedure. A test case 
is defined by a procedure name and its parameters, i. 
e., test case parameters. Also, state of environment may 
be a factor of defining a test case. The test drivers use 
the test case parameters and execute test cases on the 
SUT 3 to verify the procedure interface 4. 
[0033] The test driver generator 30 generates the test 
driver sources which, once compiled into the test drivers 
by the implementation language compiler 18. fulfil func- 
tions to initialize the procedure interface 4, prepare input 
values, call tested procedures with test case parame- 
ters, and receive test procedure results and analysis of 
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the test results. In the general case, the test driver sourc- 
es are complex programs. 

[003.4]. It is preferable that the test driver generator 30 
generates the test driver sources that, once compiled 
into the test drivers, do not only pass some previously 
generated test case parameters to the SUT 3, but also 
control the state of the SUT 3. If the SUT state violates 
some conditions of the test, the test drivers do not supply 
test parameters to the procedure interface 4. 
[0035] As the formal specifications are implementa- 
tion independent, the generated test driver sources and 
test case parameter sources are also implementation 
independent. 

[0036] The test driver generator 30 preferably com- 
prises a basic driver generator 34 and a script driver 
generator 36. The basic driver generator 34 analyses 
the formal specifications, and generates the basic driver 
sources comprising programs in implementation-inde- 
pendent language. The basic driver sources are used 
for generating a basic driver in implementation lan- 
guage. The basic driver is a test driver for a target pro- 
cedure 4. The basic driver checks whether pre-condi- 
tions for the target procedure 4 hold for a given tuple of 
input parameters, calls the target procedure 4 with the 
given tuple of input parameter, records corresponding 
output parameters, and assigns a verdict on the correct- 
ness of the target procedure execution results. The ba- 
sic driver preferably also collects information necessary 
to estimate test coverage or investigate reasons for a 
fault, as described below. 

[0037] The script driver generator 36 generates script 
driver sources which describe sequences of calls to the 
basic driver with different test case parameters. The 
script driver sources are used for generating script driv- 
ers in implementation language. A script driver is a test 
driver for a target procedure or a set of target proce- 
dures. A script driver reads test options, generates sets 
of input parameters based on test options, and calls a 
basic driver with some set of input parameters. A script 
driver may also perform extra checking of the correct- 
ness of the target procedure execution results and as- 
signs a verdict. A script driver may also check whether 
the test coverage is complete, and if not, it may continue 
to generate sets of input parameters and call the basic 
driver with this tuple. 

[0038] The present invention may be suitably applied 
to generation of a verification system for arbitrary pro- 
cedure interface of arbitrary systems. For example, the 
present invention is suitably applied to generate a veri- 
fication system for procedure interfaces of a kernel of a 
Support Operating System (SOS) for a Digital Multiplex- 
ing Switch (DMS). The invention is hereinafter de- 
scribed mainly for verification of SOS kerne! interfaces, 
but it is not limited to this application. 

Generating formal specifications 

[0039] The generation of the formal specifications of 



the procedure interfaces is further described with refer- 
ence to Figures 3 and 4. 

[0040] The means 12 for generating specifications 
first provides a function (F12) for defining procedure in- 

5 terfaces of the SOS kernel (S12). 

[0041] As shown in Figure 4, SOS 40 has SOS kernel 
42 and SOS utilities 44. SOS 40 supports applications 
46. SOS 40 is written using Nortel Networks Limited's 
proprietary programming language called Protel, which 

io is an example of the implementation, or target, lan- 
guage. 

[0042] The SOS Kernel 42 comprises a plurality of 
procedures. The procedure interface defining function 
(F12) categorises the procedures of the SOS Kernel 42 
15 into two groups: one group for those depending on im- 
plementation of SOS 40, and the other group for those 
independent from implementation of SOS 40. The pro- 
cedure interface defining function (F12) then defines 
procedure interfaces to consist of procedures that are 
20 implementation independent. The defined procedure in- 
terfaces form a Kernel Interface Layer (K I L). KIL43does 
not depend on implementation and, in particular, on 
hardware special features of SOS 40. The procedure 
interfaces of KIL 43 are defined such that each proce- 
ss dure in KIL 43 performs one and only one service. No 
two procedures provide the same service. Thus, KIL 43 
comprises minimal and orthogonal procedures needed 
by upper layers of SOS 40 and applications 46. KIL 43 
hides internal data structures and implementation de- 
30 tails of the SOS kernel 42. 

[0043] Based on the defined procedure interfaces of 
KIL 43, the means 12 for generating specifications pro- 
vides a function (F14) for developing implementation in- 
dependent description of the procedure interfaces of KIL 
35 43 (S14). 

[0044] The description developing function (F14) rig- 
orously describes functionality of the procedure interfac- 
es of KIL 43. 

[0045] The implementation independent description 
40 may be developed using reverse engineering. The basic 
idea of the reverse engineering approach is a gradual 
"upward in g" of data representation in defined imple- 
mentations. "Upwarding" is increasing the level of ab- 
straction. 

45 [0046] For example, as shown in Figure 5, it may be 
developed using source code 50 of the SOS kernel 42. 
The source code 50 is in the implementation language 
of SOS 40. The source code 50 is compiled into imple- 
mentation independent language to generate a prime 

50 specification, i.e., implementation independent descrip- 
tion 54. It is preferable to use an implementation inde- 
pendent language compiler 53 to carry out this compil- 
ing process automatically. 

[0047] The implementation independent description 
55 may also be developed from documents or other infor- 
mation of the SOS Kernel 42. 

[0048] As shown in Figure 3, the means 12 then pro- 
vides a function (F16) for deriving formal specifications 
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of K! L 43 from the implementation independent descrip- 
tion (S16). In the example shown in Figure 5, the level 
of abstraction of the prime specification 54 is increased 
to generate a formal specification 56. This abstraction 
process 55 may be carried out manually. s 
[0049] It is preferable to use Rigorous Approach to In- 
dustrial Software Engineering (RAISE) to generate for- 
mal specifications. RAISE Specification Language 
(RSL) is suitable to write formal specifications. RSL is 
supported by commercial tools for syntax and semantics 1 o 
checking, such as an EDEN-sintaxicaly oriented editor, 
a RAISE to ADA compiler, and a RAISE to C++ compiler. 
[0050] Other RAISE features, e.g., axiom, algebraic 
specifications and channels may be used in semiformal 
considerations and explanations. 15 
[0051] Also, it is preferable to use model-oriented 
specification in implicit form as the main form of speci- 
fication. The implicit form describes a target procedure 
using pre-conditions and post -conditions of the target 
procedure. 20 
[0052] The means 1 2 for generating specification may 
comprise a tool or a set of tools for providing above de- 
scribed functions for aiding a specifier to manually or 
semiautomatically generates the specifications. An ex- 
ample of such tools is the implementation independent 25 
language complier 53 as described above. 
[0053] It is preferable to classify procedure interfaces 
of the target SUT by using the specifications. The fol- 
lowing classification of procedures of a procedure inter- 
face is suitably used for generating a verification system 30 
for the procedure interface. The procedure interface 
classes include five main classes of procedures and 
some extensions of classes including procedures tested 
in parallel and expected exceptions. The classes are or- 
ganized hierarchically. The first class establishes the 35 
strongest requirements. Each following class weakens 
the requirements. The requirements for the five classes 
are as follows: 

Kl ND_1 : The input is data that could be represented 40 
in literal (textual) form and can be produced without 
accounting for any interdependencies between the 
values of different test case parameters. Such pro- 
cedures can be tested separately because no other 
target procedure is needed to generate input test 45 
case parameters and analyse the outcome of the 
tests. 

KIND_2: No interdependencies exist between the 
input items, i.e., values of input test case parame- 
ters. The input does not have to be in literal form, so 
Such procedures can be tested separately. Exam- 
ples of this class include procedures with pointer 
type input parameters. 

KIND_3: Some interdependencies exist, however, 
separate testing is possible. Examples of this class 55 
include a procedure with two parameters in which 
the first one is array and the second one is a value 
in the array. 



KIND_4: The procedures cannot be tested sepa- 
rately, because some input test case parameters 
can be produced only by calling another procedure 
from the group and/or some outcome of tests can 
be analysed only by calling other procedures. Ex- 
amples of this class include a procedure that pro- 
vides stack operations and that receives the stack 
as a parameter. 

KIND_5: The procedures cannot be tested sepa- 
rately. Part of the input and output data is hidden 
and the user does not have direct access to data. 
Examples of this class include instances of Object- 
Oriented classes with internal states; and a group 
of procedures that share a variable not visible to the 
procedure user. 

[0054] Exception raising extension of API classes: 
The specific kind of procedures raise exceptions as a 
correct reaction to certain input test case parameters. 
Examples of this class include a procedure that is sup- 
posed to raise an exception after dividing by zero. If zero 
received as an input parameter, then this procedure 
must not return any return code. 

Generating test sources 

[0055] The generation of the test sources is further 
described referring to Figure 6. Figure 6 shows an ex- 
ample of the test generation for a KIL 43 using RAISE 
as implementation independent language. 
[0056] The test source generator 1 00 comprises a ba- 
sic driver generator 102, script driver generator 104 and 
test case parameter generator 106. In this example, the 
test source generator 100 uses UNIX, and the target 
SOS kernel 42 uses target language. The formal spec- 
ifications 110 are generated in RSL. Accordingly, the 
test source generator 100 uses an RSL-target language 
complier 108 as an implementation language compiler. 
[0057] The main source of the test source generation 
is the RAISE specifications 110. The RAISE specifica- 
tions 110 are written in RSL. The RAISE specifications 
110 may be those generated by the means 12 for gen- 
erating specifications shown in Figure 1 or those stored 
in the repository 116. 

[0058] The basic driver generator 102 receives the 
specifications 110. The basic driver generator 102 is a 
tool for generating basic driver sources, i.e., RSL basic 
drivers 1 03. The RSL basic drivers 1 03 are testing pro- 
cedures in RSL The basic driver generator 102 exe- 
cutes analysis of the RAISE specifications 110. Based 
on the analysis results, the basic driver generator 102 
generates testing procedure programs comprising the 
RSL basic drivers 103. That is, the basic driver genera- 
tor 102 generates, as the RSL basic drivers 103, pro- 
grams for checking input test case parameters, calling 
tested procedures, tracing and analysing the test re- 
sults, assigning a verdict of the outcome, and outputting 
trace information. 
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[0059] The basic driver generator 1 02 preferably also 
generates source for test case parameter generation 
1 09. The source 1 09 for test case parameter generation 
preferably includes source for partition analysis, as de- 
scribed below. 

[0060] The results 1 03, 1 09 of the basic driver gener- 
ator 102 are fully completed in RSL sources. RSL gen- 
erated sources do not require any customization as they 
are implementation independent. 
[0061] The RSL basic drivers 103 generated by the 
basic driver generator 1 02 are compiled by the RSL-tar- 
get language compiler 108 into basic drivers 122 in the 
target language. The basic drivers 122 comprise target 
language procedures. Other than the language used, 
the RSL basic drivers 103 and the basic driver 122 in 
the target language are the same. 
[0062] For each procedure in KIL 43, one basic driver 
122 is generated. Each basic driver 122 provides direct 
call of a target procedure in KIL 43, and provides com- 
mon facilities to test the target procedure. That is, each 
basic driver 1 22 takes input test case parameters for Kl L 
43, and checks pre-conditions of the target procedure. 
If the pre-conditions are correct, the basic driver 122 
makes the call of the target procedure; and checks post- 
conditions of the target procedure. 
[0063] The basic drivers 122 may carry out test result 
analysis by recording execution outcomes and compar- 
ing them with required outcomes. The basic drivers 122 
may provide the result of the analysis as a verdict. The 
verdict may be either "passed" or "failed". The "passed" 
verdict means that no error is detected. The "failed" ver- 
dict means that an error is detected. 
[0064] The basic drivers 122 may have a test oracle 
to automatically perform the analysts of the test out- 
come. The test oracle is a program that assigns a verdict 
on the correctness of outcome for the target procedure. 
The test oracle is similar to post -conditions. Both the test 
oracle and the post-conditions have Boolean functions. 
They have the same parameters, and return "True" if the 
target procedure produces a correct result and "False" 
otherwise. Accordingly, the test oracles can be generat- 
ed once the post-conditions are generated. 
[0065] The test result may depend on the SOS state 
and the history of SOS functioning. In order to fulfil its 
function, each basic driver 122 preferably also gener- 
ates programs to support a model of SOS state. The 
model is used to check acceptability of test case param- 
eters in different contexts and to analyse correctness of 
test results. 

[0066] The test case parameter generator 106 re- 
ceives the source for test case parameter generation 
109 from the basic driver generator 102. Then, the test 
case parameter generator 106 generates test case pa- 
rameter sources, i.e., RSL test case parameters 107. 
The RSL test case parameters 107 may be constant ar- 
rays or programs. The test case parameter programs 
are also fully completed RSL sources. 
[0067] The test case parameter generator 106 may 



also generate test case parameter sources from the 
specifications. 

[0068] The RSL test case parameters 107 are com- 
piled into test case" parameters 126 by the RSL-target 

s language compiler 108. The test case parameters 126 
are input parameters for procedures under testing. 
Therefore, they are used for basic driver procedures. 
The test case parameters 1 26 may include only numeric 
and/or boolean input parameters. For example, a KIL of 

io SOS includes about 140 procedures which need only 
such input parameters. These procedures are called 
KIND_1 procedures, as described above. 
[0069] The script driver generator 104 receives the 
RAISE specifications 110, and generates script driver 

is sources, i.e., RSL script drivers 105. The RSL script 
drivers 105 are compiled by the RSL-target language 
compiler 108 into script drivers 124 in the target lan- 
guage. Other than the language used, and the RSL 
script drivers 1 05 and the script drivers 1 24 in the target 

20 language are the same. The script drivers 124 are the 
upper level of the basic drivers 1 22. 
[0070] Each RSL script driver 103 is a program for 
testing of a procedure or a group of procedures. It is a 
sequence of target procedures calls. The sequence may 

25 have serial or parallel composition. The sequence may 
have iterations. The RSL script drivers 103, once com- 
plied into the script drivers 124 by the complier 108, re- 
alize a scenario or script of testing. 
[0071] The script driver generator 104 generates, as 

30 the RSL script drivers 105, programs to realize the se- 
quences of procedure execution with different test case 
parameters. The script driver generator 104 generates 
the RSL script drivers 105 to have no direct interaction 
with target procedures. That is, the RSL script drivers 

35 105, once complied into the script drivers 124, call the 
basic driver 122. One or more RSL script drivers 105 
may be written to be called by procedures which function 
as suppliers of test case parameters 1 26, or procedures 
that allow a system operator to control a procedure 

^o group testing. 

[0072] The script driver generator 1 04 may also gen- 
erate programs to check the verdicts of the basic drivers 
122. The script driver generator 104 may also generate 
programs to assign script driver own verdicts based on 

^5 the basic driver verdicts. 

[0073] It is preferable that the script driver generator 
104 uses script driver skeletons 112 in addition to the 
specifications 110. The script driver skeletons 112 de- 
scribe general scheme of script drivers. That is : each 

50 script driver skeleton contains an algorithm of a script 
driver. The script driver skeletons 112 are specific to 
each kind of procedure interface. 
[0074] Each script driver consists of declarations and 
a body. The declarations include import of the procedure 

55 under test and its data structure definitions and/or import 
of all data and types used in the specifications. The dec- 
larations are generated automatically based on the list 
of procedures under test and their specifications 110. 
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The body of a script driver begins with the script driver 
option parsing. The options, as parameters of the script 
driver as a whole, determine the depth of testing, e.g., 
the level of test coverage criteria, and some specific da- 
ta like interval of values, duration of testing. s 
[0075] In the example shown in Figure 6, in order to 
generate an RSL script driver 1 05, the script driver gen- 
erator 1 04 uses one of the skeletons 1 1 2 and the RAISE 
specifications 110. Union of the specifications 110 and 
skeletons 112 forms formal description of test suite io 
sources. This formal description may be considered as 
a test suite specification. The test suite specification al- 
lows the generator 100 to define test coverage require- 
ments, schemes of script drivers, and algorithm for 
checking target procedure behaviours. is 
[0076] The script driver skeletons 1 1 2 for a new target 
SUT may be manually developed or received from the 
repository 116. Before testing starts, the verification sys- 
tem carries out some initialization. For example, before 
testing write/read procedures, the verification system 20 
opens a file. Such initializations are written manually. Af- 
ter initialization is finished, the main part of the script 
driver begins. 

[0077] In addition to specifications 1 1 0 and skeletons 
1 1 2, the script driver generator 1 04 may also use some 2s 
supplement sources, such as some instances of test 
case parameters values. 

[0078] The script driver generator 104 may also use 
procedures that convert values derived from the RAISE 
specifications 1 10 into value formats used by the current 30 
version of SOS kernel 42. Because the specifications 
110 are implementation independent, correspondence 
between the specifications 110 and implementation da- 
ta structures is separately described. Thus, it is prefer- 
able to use some means for associating abstract objects 35 
with implementation objects. Some target language pro- 
cedures convert data from their representation in imple- 
mentation to and from their representation in the test 
suite 120. Such target language procedures may be 
used as the associating means. The target language 40 
procedures use post-conditions of the procedure under 
test. The target language procedures may be manually 
developed. 

[0079] These additional sources including manually 
written skeletons may be called "manually developed 46 
components". The size of manually developed compo- 
nents is not large compared to the automatically gener- 
ated components in the verification system generator 
100. 

[0080] For KIND_1 procedures, full automation of test so 
generation is possible. All other kinds generally need 
some additional effort for writing manually developed 
components. The effort gradually grows from KIND_2 
to KIND_5. The extensions require more effort than the 
corresponding kinds themselves. Complexity and effort ss 
for the development of manually developed compo- 
nents is usually caused by the complexity of the script 
driver generation and debugging. All script drivers for 



different classes of procedures have similar structure. 
The main distinction is the distribution between auto- 
matically generated components and manually devel- 
oped documents. The KIND_1 script driver is generated 
fully automatically, Kl ND_2 script driver is generated al- 
most automatically and so on. 

[0081] The scheme of a script driver is further de- 
scribed in more detail using an example of a KIND_5 
script driver. 

[0082] The KIND_5 script driver realizes a general al- 
gorithm for traversing an abstract Finite State Machine 
(FSM). This algorithm passes all states and all possible 
transitions between the states. Each transition corre- 
sponds to an execution of a procedure under test. 
[0083] The algorithm of a script driver is related to the 
specification and does not depend on the implementa- 
tion details outside the specification. The script driver 
algorithm does not have direct descriptions of the ab- 
stract FSM. The verification system generator 100 
avoids use of direct descriptions because direct speci- 
fication of the FSM requires extra efforts to generate. 
[0084] Instead of a direct specification of FSM, the 
verification system generator 100 uses indirect, virtual 
representation of FSM. Such representation includes a 
function-observer and a function -iterator. The function- 
observer calculates on the fly the current state in the 
abstract FSM. The function-iterator selects a next pro- 
cedure from the target procedure group, and generates 
a tuple of the input parameter values for this procedure. 
[0085] The KIND_5 script driver algorithm is de- 
scribed in more detail. For example, a case of testing a 
procedure group is considered. After passing several 
FSM states : i.e., some target procedures have been 
called, the next transition is being made. This elemen- 
tary cycle of testing starts by calling a function-iterator 
that selects the next procedure from the target proce- 
dure group, and prepares a tuple of input test case pa- 
rameter values for this target procedure. If the function- 
iterators have managed to generate a new and correct 
tuple without violation of pre-conditions, then the script 
driver calls a corresponding basic driver with the tuple 
as actual test case parameters. 

[0086] When the basic driver returns a verdict, the 
control script driver checks the verdict assigned by the 
basic driver. If the verdict is "False", i.e., an error has 
been detected, the script driver produces corresponding 
trace data and finishes. If the verdict is "True", i.e., the 
elementary test case passed, the script driver calls the 
function-observer. The function-observer then calcu- 
lates a current state, logs the state and transition, and 
continues to traverse FSM. 

[0087] Thus, all possible states and test the proce- 
dures with all needed sets of input parameters may be 
obtained. FSM is used here as a guideline to pass 
through all states the needed number of times. 
[0088] As described above, the script drivers are pref- 
erably composed following the requirements of the cor- 
responding skeletons. In this embodiment, overall, the 
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verification system generator 100 uses five skeletons 
needed for serial testing of API KIND-1 through KIND_5 
and one skeleton for parallel testing^ Based on a corre- 
sponding skeleton and the list of targefprocedures and 
specifications, the verification system generator 100 
generates a script driver template for each class. A 
KIND_1 template is a ready-to-use program. The tem- 
plates for the other kinds include several nests with de- 
fault initiators and iterators. If a test designer does not 
need to add or improve anything in the nests, the tem- 
plate can be compiled by the RSL-target language com- 
piler 108 and executed as a script driver 124. This situ- 
ation is typical for a KIND_2 procedure interface. For 
other kinds, a test designer usually adds some specific 
initiators and iterators as RSL supplement 115. The test 
designer defines FSM state observer for the script driv- 
ers of KIND_4 and KIND_5. 

Fo rma lion of skeie tons 

[0089] The development or formation of skeletons 
112 is now described referring to Figure 7. 
[0090] According to the present invention, skeletons 
112 are developed by decomposition of test suites 
(602). Every large system consists of subsystems. A 
structure that exposes the hierarchical construction of 
the subsystem is called its decomposition. Test suites 
have generated and manually-developed components, 
and these components are exposed by the decomposi- 
tion. 

[0091] Standard schemes of the procedure testing 
are then determined based on the decomposition (604). 
As described above, the standard schemes include the 
five kinds, Kind_1 to Kind_5. Each scheme is defined 
by a skeleton of the test suite. 

[0092] For each separate scheme of procedure test- 
ing, skeleton description is developed in the skeleton 
definition language (606). The skeleton definition lan- 
guage is analogous to the macro language of an ad- 
vanced programming language. The main difference 
between the skeleton language and the macro language 
is that the skeleton language does not use the macro 
definitions and macro calls. In this sense, the skeleton 
language functions as a directive for control of transla- 
tion. 

[0093] The skeleton definition language consists of 
the following parts: 

invariant test suite parts written in some test suite 

implementation language; 

skeleton parameter identifiers; 

repetitors describing the repetitive parts of the test 

suite; 

variant descriptors of the test suite variants: and 
slot descriptors for the manually-developed compo- 
nents of the test suite. 

[0094] Parameters of the skeleton may be text strings, 



integers, arrays of text strings or arrays of integers. Ar- 
rays of text strings or integers may be one dimensional 
arrays or multi dimensional arrays. Skeleton parameter 
identifiers identify the type of parameters. ' 
s [0095] Repetitors and variant descriptors are written 
in the macro-language of the skeleton definition lan- 
guage. This macro-language may be a subset of the C 
language. For example, a repetitor may be the following 
construct: 

10 for (i=1; k=upbi;. i+=l) ^repetitive part including 

PAR[i)>} 

where upbi is the integer parameter of the skeleton, and 
PAR is the one-dimensional array of text strings. 
[0096] Slot descriptors provides slots in skeletons. 

f5 Some slots are filled in automatically, and another slots 
are filled in manually. Each manually filled in slot has the 
rigorously defined semantics. The slot descriptor sup- 
ports the rigorously defined semantic. Each slot descrip- 
tor consists from two slot boundaries, slot identifier hav- 

20 ing an arbitrary length string and a default slot filler. The 
semantic of the clot filler is defined by its identifier. No 
formal semantic description is provided. 
[0097] Finally, the skeleton description written in the 
skeleton definition language is then transformed into a 

25 skeleton tool (608) The skeleton tool, or the skeleton, 
facilitates generation of the test suite of the selected 
scheme. The translation of the skeleton description may 
be carried out by transforming the skeleton description 
into the source code of the skeleton tool for generating 

30 the template of test suite. The source code may be C- 
code. During transformation, each part of the skeleton 
definition is transformed into the corresponding part of 
the skeleton tool. That is, the invariant test suite parts 
of the skeleton description are transformed into constant 

35 text strings. The skeleton parameter identifiers are 
transformed into the definition of the variable. The repet- 
itors and the variant descriptors become the part of the 
source code of the skeleton tool as they are written in 
the macro-language of the skeleton description. The slot 

^o descriptors also transformed into the constant text 
strings. 

[0098] Formal requirements may be written in the 
skeleton for each slot for manually-developed compo- 
nents. 

45 [0099] Figure 9 shows a skeleton forming system 700 
according to an embodiment of the present invention. 
The system 700 comprises a decomposer 702, a skel- 
eton describer 704 and a skeleton transformer 706. 
[0100] The decomposer 702 decomposes test suites 

50 (602). It has functions 710-720 for identifying specific 
parts of test suites. The skeleton describer 704 develops 
skeleton description (606). It has functions 730-740 for 
creating parts of skeleton description based on the iden- 
tified parts of the test suites. 

55 [0101] As shown in Figures 8 and 9, the decomposer 
702 has an invariant test suite parts identifier 710.that 
identifies invariant test suite parts (610). Based on the 
identified invariant parts, the invariant test suite descrip- 
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tion creator 730 of the skeleton describer 704 creates 
invariant test suite description (630). 
[0102] A skeleton parameter identifier 712 of the de- 
composer 702 identifies skeleton parameters (612). 
Based on the identified parameters, an identifier creator s 
732 of the skeleton describer 704 creates skeleton pa- 
rameter identifiers (632). 

[0103] Similarly, identifiers 714-720 of the decompos- 
er 702 identify from the decomposed test suites, repet- 
itive parts, variants, manually developed components 10 
and automatically generated components, respectively 
(614-620). Based on the identified repetitive parts, var- 
iants, manually developed components and automati- 
cally generated components, creators 734-740 of the 
skeleton describer 704 creates repetitors, variant de- is 
scriptors, first slot descriptors and second slot 
descriptors , respectively (634-640). 
[0104] The order of identifying those features do not 
need to follow the above order 

[0105] The transformer 706 transforms the skeleton 20 
description into a skeleton tool (608). The skeleton tool 
uses a file containing the skeleton parameter actual val- 
ues and the components for the slot filling. 
[01 06] Completed test suite sources will be obtained 
as a result of the execution of the skeleton tool. For ex- 25 
ample, as shown in Figure 10, the skeleton generated 
by the skeleton transformer 706 is used as an input for 
generation of a tool, template generator 750. The pro- 
cedure specification generated by the means for gener- 
ating specification 1 2 (Figure 1 ) is also used as an input 30 
for generation of the template generator 750. The tem- 
plate of the test suite is obtained as the result of execu- 
tion of the template generator 750. The template con- 
tains the slots for manually written parts. A tool, template 
filler 754, receives the test suite template and a file with 35 
manually written parts of the test suite as its input, and 
generates the completed test suite sources, which will 
then be compiled into the test suite 22 as described be- 
low. 

[0107] In an example, during test suite development 40 
in accordance with the present invention, the size of the 
manually-developed components can be less than 20% 
of the total test suite size. 

[0108] The invention decreases efforts required for 
the test suite development, and increases reliability of 45 
the resultant test suites because of simplifications and 
formalizations of the requirements to the manually-de- 
veloped components. 

Compilation of test sources so 

[0109] Returning back to Figure 6, in the generator 
1 00, all kinds of generation by generators 1 02, 1 04 1 06 
produce results 103, 105, 107, 109 in RSL This means 
that "front end" of specification and verification technol- 55 
ogy is implemented in implementation language inde- 
pendent form. All generators 102, 104, 106 can produce 
the components 122, 124, 126 of the test suites 120 for 



systems implemented in arbitrary programming lan- 
guages. 

[0110] Compilation of generated sources 103, 105, 
107 by the RSL-target language compiler 108 may be 
carried out when generation of all sources 1 03, 1 05, 1 07 
is completed. The RSL-target language compiler 108 
translates executable subsets of RSL language into pro- 
grams in the target language. Thus, the RSL-target lan- 
guage compiler 108 restricts RSL. These restrictions 
are typical for all RSL language compilers. For example, 
the RSL-target language compiler 1 08 does not treat ex- 
plicit definitions of constants if the user does not define 
the concrete constant value but only defines limitations 
that restrict constant field of values. 
[0111] The RSL-target language compiler 108 is im- 
plementation-language dependent. 
[0112] The result of the RSL-target language compiler 
108 is generally a group of complete target-language 
sections. This is a part of the target language module 
that consists of a few sections. For obtaining a target 
language program which is ready to execute, some tar- 
get language sections with interface descriptions may 
be produced. Interfaces or behaviour of some proce- 
dures from SOS are written once and do not need to be 
rewritten repeatedly. The target language sections with 
interface descriptions may be produced manually. 
These target language sections may be called target 
language supplement 114. 

[0113] In order to correctly use different generation/ 
compiling tools, it is preferable to know interdependen- 
ces between modules of specifications and between re- 
suits of generation/compiling, i.e., the target language 
sections, and other target language modules/sections 
that were manually developed or had been produced by 
other tools. These interdependences may be repre- 
sented by a graph. The complexity of such a graph of 
interdependencies depends on the size of the proce- 
dure interface under test. 

[011 4] For example, currently KIL consists of over 560 
procedures divided into over 30 subsystems. For each 
subsystem, there exists, at least, a basic driver module, 
and as a whole there exist about 200 script driver mod- 
ules. For each RSL driver, at least one target language 
module is generated and stored. Besides, the target lan- 
guage modules consist of a few sections and each sec- 
tion is stored in a separate file. As a whole, KIL requires 
over 10,000 files. In order to facilitate use of test gener- 
ation/compiling tools, it is preferable to provide a work 
"manage" utility, as described later. 
[0115] The basic drivers 122 invoked by the script 
drivers 124 are generated fully automatically. The only 
manually developed components called from basic driv- 
ers 122 are data converters of the RSL-target language 
compiler 108. As mentioned above : the converters 
transform the model data representation into the imple- 
mentation representation and vice versa. A model rep- 
resentation is distinguished from the implementation 
one by the level of abstraction. For example, models 
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may use "infinite" representation of integers, maps, re- 
lations, and other data structures suitable for specifica- 
tion. Sometimes model representation is very similar to 
the implementation one. Tnthis case, such transfomna- 
tion is done by a standard translation algorithm of the s 
specification language into the implementation lan- 
guage. 

[0116] The verification system generator 100 is suita- 
bly used for generating a verification system for a con- 
tinual evolving SUT. SOS may be evolved in accordance io 
with its life cycle. During evolution cycle, requirements, 
interfaces or behaviour of some procedures from the 
SOS kernel, and implementation of SOS are repeatedly 
modified. For each new version of SOS, it is necessary 
to develop a new version of verification system. There- is 
fore, it is beneficial to automate process of regeneration 
of the verification system. 

[0117] Life cycle of test suites 120 generated by the 
verification system generator 100 replicates life cycle of 
the SOS Kernel 42. Usually, only a few interfaces or be- 20 
haviour of some procedures from the SOS kernel are 
modified. The verification system generator 100 pro- 
vides a possibility to re-specify modified interfaces or 
behaviour of some procedures from the SOS kernel and 
then re-generate test suites 1 20, and in doing so to pro- 25 
vide re-use of old manually developed components. 
Thus, the verification system generator 100 can auto- 
mate test suites regeneration. Therefore, existence of 
manually developed components does not decrease ac- 
tual level of automation of the verification system gen- 30 
e ration. 

[0118] To support automatic regeneration of test 
suites 120, the verification system generator 100 pref- 
erably stores in the repository 116 all manually devel- 
oped components developed for generating the test 35 
suites 120 separately from automatically generated 
components. The manually developed components 
supplement automatically generated components. 
Therefore, the process of the test suites components' 
manual development may be called "supplement". 40 
Thus, the verification system generator 100 may use 
two kind of sources for generating test sources: formal 
specifications and some supplement sources. As auto- 
matically generated and manually developed compo- 
nents of the verification system generator 1 00 are stored 45 
separately, no manual changes in automatically gener- 
ated components are needed. Therefore, the verifica- 
tion system generator 100 can eliminate need of cus- 
tomizing automatically generated files for each regen- 
eration of the test suites 120. so 
[0119] To estimate effort for generating verification 
system, a volume of modified interfaces or behaviour of 
some procedures from the SOS kernel is first estimated. 
When no interface is modified during SOS evolution, 
then no test (re)generation is needed. In that case, only * ss 
realization, i.e., implementation, of SOS is modified. 
Therefore, previous specifications 110 and previous test 
suites 1 20 can be used for validation of the new Ki L. 



[0120] When some interfaces or behaviour of some 
procedures from the SOS kernel are modified or added 
during SOS evolution, then corresponding specifica- 
tions 110 heed to be mod ifiedrWhen~interf ace "data 
structures are modified, in addition to specifications 110, 
some conversion procedures in the target language also 
need to be (re)developed. Those target language con- 
version procedures may be manually developed. In any 
case, some reasons for test plan modification may arise. 
For example, these modifications may be caused by 
wishes to increase amount of tests, decrease time of 
testing, to check correlation of some features for parallel 
execution and so on. In those cases, some manual mod- 
ification to manually developed components may be 
needed. When manual modifications are completed, a 
test designer can automatically generate new test suites 
120 for validation of the new SOS kernel by using the 
verification system generator 100. 
[0121] In a simple case, it may suffice to modify the 
specifications 110 of types of pre-condition or post-con- 
dition of a target procedure. When new modification of 
procedure behaviour does not imply/impinge on behav- 
iour of other procedure, the generator 100 needs only 
to regenerate a basic driver for verification of the modi- 
fied procedure. In a complicated case, the generator 
100 may need to regenerate totally new test suites in- 
cluding new basic drivers and script drivers. What vol- 
ume of test suite modification is required depends on 
dependencies inside of the specifications 110 and be- 
tween separate parts of the specifications 110 and test 
suites components 122-126 generated from these 
parts. Existing "manage" utility may be used which au- 
tomates regeneration and recompiling of new test 
suites, as described later. 

[0122] In order to port a test suite 1 20 generated by 
the verification system generator 100 from one imple- 
mentation language platform to another, the data con- 
verters need to be rewritten and a new RSL to imple- 
mentation language compiler needs to be provided. Al- 
so, a new run-time support system for the test suites 
with new test bed functions needs to be provided. 
[01 23] It is preferable that the verification system gen- 
erator 100 also generates data for test coverage esti- 
mation and test plan design. The data is preferably kept 
in the repository 116. 

[0124] Test coverage measures the completeness of 
testing. Sometimes, test coverage is presented as per- 
centage of checked test situations. Sometimes, it is a 
list of test situations that have been or should be 
checked by the test suites. Test coverage requirements 
present all possible test situations that must be covered 
by test suites execution. If test suites 120 meet the re- 
quirements, then "exhaustive" or "100%" test coverage 
is gained. 

[0125] There is a difference between test coverage 
estimation for source code and for specifications. In the 
case of source code, a test situation is associated with 
a statement, branch of path in control flow graph of a 
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program. In the case of specifications, the specifications 
are represented as logical expressions, i.e., Boolean ex- 
pressions. Thus, test situations are associated with 
branches and disjuncts of Boolean expressions. There- 
fore, by using test situations for specifications, it is pos- 
sible to define test coverage requirements for arbitrary 
specifications. This allows uniform notation for descrip- 
tion and accounting of the test situations, coverage re- 
quirements and obtained coverage. 
[0126] The test suites are generated based on the 
specifications. The specifications are implementation 
independent. Thus, the test coverage is preferably 
measured by means of an implementation independent 
way. For this purpose, the verification system preferably 
uses test coverage criteria which are based on the spec- 
ifications. 

[0127] In complex SUTs, "all test situations" may not 
be covered. Accordingly, it is preferable to group similar 
test situations in classes. In this case, exhaustive cov- 
erage may represent coverage of all classes of test sit- 
uations. Test situations and their classes may be iden- 
tified and classified based on implementation source 
code or some external descriptions of the procedures 
under test. When a so-called "black box" approach is 
taken to test SUTs, test situations and their classes are 
identified and classified based on knowledge of descrip- 
tions of the procedures under test. 
[01 28] The test coverage criterion is a metric defined 
in terms of implementation or specification. The most 
well known test coverage criteria in terms of implemen- 
tation are: 

Class 1 - all statements are passed; and 
Class 2 - all branches are passed. 

[0129] In the case of using the specifications for test 
coverage criteria definition, the so-called domain testing 
approach is preferably used. The whole input space is 
partitioned into areas by the basic driver generator. 
Each area corresponds to a class of equivalence. 
[01 30] The source for test case parameter generation 
109 generated by the basic driver generator 102 prefer- 
ably includes source for the partition analysis. The par- 
tition determines the choice of one of the test generation 
techniques applicable to a procedure interface or an in- 
terface of a procedure group. The source for partition 
analysis includes a list of test situation classes that rep- 
resent test coverage requirements, and initial data for 
partition analysis. The source for partition analysis is 
used to generate test case parameters 126. 
[0131] The partition may be derived from the specifi- 
cations that describe requirements on input and proper- 
ties of outcome for target procedures. Both the require- 
ments and properties are represented in pre-conditions 
and post-conditions of formal specifications in implicit 
form. Accordingly, the test coverage estimation can be 
carried out based on the implicit specifications. In this 
example, the average percentage of the test coverage 



of the verification system generated by the generator 
100 for SOS KIL is 70% to 100% of statements in the 
implementation. 

[0132] Furthermore, there are two levels of the test 
5 coverage criteria. The first one is the coverage of all 
branches in post-conditions. The second one is the cov- 
erage of all disjuncts, i.e., elementary conjunctions, in 
the Full Disjunctive Normal Form (FDNF) representation 
of the post-condition while taking into account the pre- 
10 condition terms. The verification system generator 100 
allows automatic partitioning in terms of specification 
branches and FDNF. It is preferable to calculate acces- 
sible FDNF disjuncts and remove the inaccessible FD- 
NF disjuncts using pre-condition design. 
15 [0133] Monitoring of obtained test coverage is prefer- 
ably conducted on the fly by script drivers 124. Based 
on this data, the script drivers 124 may tune testing pa- 
rameters and/or testing duration. 

[0134] As described above, the verification system 
20 generation consists of a sequence of steps. For a step, 
needed tools are used where possible. Some tools gen- 
erate auxiliary data, others convert and union their out- 
put and generate resulting target language code. To fa- 
cilitate usage of these tools, it is preferable to use "man- 
25 age" utility. The "manage" utility is a tool which works on 
a standard structure of UNIX directory where the tools 
and sources for verification system generations and 
generated test suites are stored. 

[0135] The "manage" utility works like UNIX "make". 

30 it analyses an interdependences graph which repre- 
sents interdependences of the test suite components, 
and searches "inconsistencies" in the graph. Each "in- 
consistency" requires to make some kind of generation 
and compiling. After the "manage" utility removes all "in- 

35 consistencies", all needed test suites components be- 
come ready to use. The "manage" utility uses a set of 
UNIX scripts and UNIX "make" files. The "manage" util- 
ity uses description of paths for all files used. It is pref- 
erable to define a "standard" directory structure to store 

40 sources and generated files for generating verification 
systems, and directories of the verification system gen- 
erator. 

[0136] While particular embodiments of the present 
invention have been shown and described, changes 

^5 and modifications may be made to such embodiments 
without departing from the true scope of the invention. 
For example, the present invention is described mainly 
using the verification system generator for verifying the 
SOS kernel. However, the present invention is suitably 

50 used for verifying a different system, such as a base lev- 
el of call processing system, and a management system 
of tree-like store for queues with different disciplines. 
The present invention is mainly disclosed using RSL 
specifications. However, natural language documenta- 

55 tion may also be used. Also, the present invention is 
mainly disclosed using UNIX. However, other operating 
systems may also be used. 
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Claims 

1 . A method for forming a skeleton tool usable for gen- 
erating a test suite for a~verification system tor ver- 
ifying a Procedure Interface Under Test (PIUT), the s 
method comprising steps of: 

decomposing existing test suites, the test 
suites having automatically generated compo- 
nents and manually developed components io 
and being written in its test suite implementa- 
tion language; 

defining one or more standard schemes of pro- 
cedure testing based on the decomposition of 
the test suites; 15 
providing skeleton description for each scheme 
in skeleton definition language; and 
transforming the skeleton description for each 
scheme into a skeleton tool for generating the 
test suite of the scheme. 20 

2. The method as claimed in claim 1 , wherein 

the decomposing step includes a step of iden- 
tifying an invariant test suite part written in the 25 
test suite implementation language; and 
the skeleton description providing step includes 
a step of creating an invariant test suite descrip- 
tion based on the identified invariant test suite 
part. 30 

3. The method as claimed in claim 1 , wherein 

the decomposing step includes a step of iden- 
tifying a skeleton parameter; and 35 
the skeleton description providing step includes 
a step of creating a skeleton parameter identi- 
fier based on the identified skeleton parameter. 

4. The method as claimed in claim 3, wherein the skel- 40 
eton parameter includes: a text string; an integer; 

or an array of text strings or integers. 

5. The method as claimed in claim 1 , wherein 

45 

the decomposing step includes a step of iden- 
tifying a repetitive part of the test suite; and 
the skeleton description providing step includes 
a step of creating a repetitor describing the 
identified repetitive part of the test suite. so 

6. The method as claimed in claim 1 , wherein 

the decomposing step includes a step of iden- 
tifying a variant of the test suite; and 55 
the skeleton description providing step include 
a step of creating a variant descriptor describ- 
ing the identified variants of the test suite. 



7. The method as claimed in claim 1 , wherein 

the decomposing step includes a step of iden- 
tifying a manually-developed component of the 
test suite; and 

the skeleton description providing step includes 
a step of creating a slot descriptor describing a 
slot for receiving a component corresponding 
to the identified manually-developed compo- 
nent of the test suite. 

8. The method as claimed in claim 1 , wherein 

the decomposing step includes a step of iden- 
tifying an automatically-generated component 
of the test suite; and 

the skeleton description providing step includes 
a step of creating a slot descriptor describing a 
slot for receiving a component corresponding 
to the identified automatically-generated com- 
ponent of the test suite. 

9. The method as claimed in any of claims 5 to 9, 
wherein the repetitor, the variant descriptor or the 
slot descriptor is written in a macro-language. 

10. The method as claimed in claim 9, wherein the slot 
descriptor provides rigorously defined semantics 
for manual slot filling. 

11. The method as claimed in any preceding claim, 
wherein the defining step includes a step of sepa- 
rating procedure testing into different types, each 
type corresponding to each standard scheme, 
based on: dependency of parameters or sequences 
of testing of a group of procedures. 

12. The method as claimed in any preceding claim, 
wherein the transforming step includes a step of 
creating a file containing values of skeleton param- 
eters and components for slot filing. 

13. A system for forming a skeleton tool usable for gen- 
erating a test suite for a verification system for ver- 
ifying a Procedure Interface under Test (PIUT), the 
system comprising: 

a decomposer for decomposing test suites, the 
test suites having automatically generated 
components and manually developed compo- 
nents and being written in its test suite imple- 
mentation language, the decomposed test 
suites being used to define one or more stand- 
ard schemes of procedure testing: 
a skeleton describer for providing skeleton de- 
scription for each scheme in skeleton definition 
language; and 

a skeleton transformer for transforming the 
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skeleton description for each scheme into a 
skeleton tool for generating the test suite of the 
scheme. 

14. The system as claimed in claim 13, wherein the 
skeleton describer includes a creator for creating 
parts of skeleton description based on the identified 
features of the test suites. 

15. The system as claimed in claim 13 or 14, wherein 
the decomposer includes a feature identifier for 
identifying particular features of the test suites. 

1 6. The system as claimed in claim 1 5, wherein the fea- 
ture identifier includes: 

an invariant test suite part identifier for identify- 
ing an invariant test suite part written in the test 
suite implementation language; 
a skeleton parameter identifier for identifying a 
skeleton parameter; 

a repetitive part identifier for identifying a repet- 
itive part of the test suite; 
a variant identifier for identifying a variant of the 
test suite; 

a manually-developed component identifier for 
identifying a manually-developed component 
of the test suite; and 

an automatically-developed component identi- 
fier for identifying an automatically-generated 
component of the test suite. 

17. The system as claimed in any of claims 13 to 16, 
wherein the skeleton describer includes: 
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wherein the scheme defining means has a function 
for separating procedure testing into different types, 
each type corresponding to each standard scheme, 
based on dependency of parameters. 

19. A method for generating test suite sources for a ver- 
ification system for verifying a Procedure Interface 
Under Test (PIUT), the method comprising steps of: 

decomposing existing test suites, the test 
suites having automatically generated compo- 
nents and manually developed components 
and being written in its test suite implementa- 
tion language; 

defining one or more standard schemes of pro- 
cedure testing based on the decomposition of 
the test suites; 

providing skeleton description for each scheme 
in skeleton definition language; 
transforming the skeleton description for each 
scheme into a skeleton tool for generating the 
test suite of the scheme 
generating a test suite template by executing 
the skeleton tool using specification of the 
PIUT; and 

filling the test suite template using manually 
written parts of the test suite to generating test 
suite sources. 

20. The method as claimed in claim 1 9, further compris- 
ing a step of compiling the test suite sources into a 
test suite in implementation language. 

21 . A computer program element comprising computer 
program code means arranged to cause a proces- 
sor to implement procedure to perform the method 
of any of claims 1 to 12 or 19 or 20. 



an invariant description creator for creating an 
invariant test suite description based on the 
identified invariant test suite part; 
a parameter identifier creator for creating a 
skeleton parameter identifier based on the 40 
identified skeleton parameter; 
a repetitor creator for creating a repetitor de- 
scribing the identified repetitive part of the test 
suite; 

a variant descriptor creator for creating a vari- 45 
ant descriptor describing the identified variants 
of the test suite; 

a first slot descriptor creator for creating a slot 
descriptor describing a slot for receiving a com- 
ponent corresponding to the identified manual- so 
ly-developed component of the test suite; and 
a second slot descriptor creator for creating a 
slot descriptor describing a slot for receiving a 
component corresponding to the identified au- 
tomatically-generated component of the test ss 
suite. 

18. The system as claimed in any of claims 13 to 17, 
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